Hooking applications are becoming increasingly popular, especially in the security and network management arts. Such hooking applications are adapted for hooking various aspects of an interface. By way of example, some of such applications are capable of hooking application program interface (API) calls.
Such API hooking is a technique where an API is modified so that subsequent invocations of a particular function transfer control to a handler. This handler may then, in turn, analyze the original API invocation, report usage of the API, modify relevant parameters, etc. Further, in the case of security applications, API hooking may serve to enable a decision as to allowing or disallowing the original API to be invoked.
Unfortunately, the control transfer instruction most commonly used during hooking is longer than a number of bytes that can be written atomically (e.g. capable of being carried out without interference from other executed threads, etc.). Because of this, a danger exists for a race condition where the replacement of the original code of the API is in an incomplete state when another thread executes this portion of the original API code. If this occurs, the results may be unpredictable and the executing thread and process may crash which, in turn, may possibly crash the operating system as well.
Additionally, the control transfer instruction most commonly used during hooking can be longer than the instruction to be replaced, so that multiple smaller instructions might be overwritten. For example, the control transfer instruction in an arbitrary example may require five bytes, but the instruction to be replaced might be only four bytes in size. A typical hook mechanism might immediately replace all five bytes required by the control transfer instruction, which in this example would replace all four bytes of the first instruction and one byte of a second instruction. A danger exists that another thread might then attempt to execute the second instruction whose bytes have been partially or entirely replaced with the bytes of the control transfer instruction.
There is thus a need for overcoming these and/or other problems associated with the prior art.